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ABSTRACT 

The goal of multi-image classification is to identify and label “similar regions” within a scene. The ability to correctly 
classify a remotely sensed multi-image of a scene is affected by the ability of the classification process to adequately 
compensate for the effects of atmospheric variations and sensor anomalies. Better classification may be obtained 
if the multi-image is preprocessed before classification, so as to reduce the adverse effects of image formation. In 
this paper, we discuss the overall impact on multi-spectral image classification when the retinex image enhancement 
algorithm is used to preprocess multi-spectral images. The retinex is a multi-purpose image enhancement algorithm 
that performs dynamic range compression, reduces the dependence on lighting conditions, and generally enhances 
apparent spatial resolution. The retinex has been successfiilly applied to the enhancement of many different types 
of grayscale and color images. We show in this paper that retinex preprocessing improves the spatial structure of 
multi-spectreJ images and thus provides better within-class variations than would otherwise be obtained without 
the preprocessing. For a series of multi-spectral images obtained with diffuse 2ind direct lighting, we show that 
without retinex preprocessing the class spectral signatures vary substantially with the lighting conditions. Whereas 
multi-dimensional clustering without preprocessing produced one-class homogeneous regions, the classification on 
the preprocessed images produced multi-class non-homogeneous regions. This lack of homogeneity is explained 
by the interaction between different agronomic treatments applied to the regions: the preprocessed images are 
closer to ground truth. The principle advantage that the retinex offers is that for different lighting conditions 
classifications derived from the retinex preprocessed images look remarkably “similew”, and thus more consistent, 
whereas classifications derived from the original images, without preprocessing, are much less similar. 

Keywords: image classification, image enhancement, multi-spectrsd, retinex, dynamic range compression 

1. INTRODUCTION 

The analysis of remote sensed imagery obtained over agricultural regions can be used for the detection and dis- 
crimination of “stressed” and “non-stressed” vegetation; this is an issue of considerable importance to the agriculture 
industry. The terms “stressed” and “non- stressed” are used in a qualitative sense to designate the relative plant 
growth over different regions of a field. The differences in growth patterns can be due to several factors including 
different agronomic treatments. Veirious algorithms for the discrimination and detection of vegetation using remote 
sensed imagery exist in the literature. One way to characterize these algorithms is by the characteristics of the 
multi-dimensional space in which they operate. For example, many users of remote sensed imagery use spectral 
signatures to characterize and identify materieJs in multi-diraensioneJ “spectral” space. The spectral signature of a 
material can be defined in the solar-reflective spectral region by its reflectance as a function of wavelength, measured 
at an appropriate spectral resolution. In other spectreJ regions, signatures of interest are temperature and eraissivity 
(Thermal Infrared TIR) and surface roughness (rad2ir).^ 

There are fundamental problems with the spectral signature approach that axe well documented in the literature. 
First, all spectral signatures are unique to the sample and the environment in which they are obtained. Second, 
the ability to distinguish spectral signatures is often complicated by natural variability for a given materi2il, spectral 
quantization of many remote-sensing systems, and modification of signatures by the atmosphere as a result of the 
image formation process.^ Therefore, even though one may wish to apply different “labels” to differentiate vegetation 
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Figure 1, System Model 

signatures, there is no guarantee that the signatures obtained by the remote sensing system will exhibit measurably 
different, or even recognizable, signatures. 

In recent years, a considerable amount of ground-based (laboratory) data have been accumulated that describe 
spectral reflectance characteristics of soils and vegetation, without the problem of atmospheric complications. It is 
difficult, however, to duplicate natural reflectance measurements under laboratory conditions. The comparison be- 
tween natural reflect 2 Uice signatures and laboratory produced signatures, therefore, becomes even more complicated. 
Furthermore, the spectral signature of vegetation also changes over the seasonal life cycle of many plants, acquiring 
a “yellow” characteristic in senescence,* with a corresponding increase in the red region reflectmce caused by a loss 
of photosynthetic chlorophyll.^ 

As an alternative to classification based on spectral signatures, multi-dimensional spectral space is transformed 
into a “feature” space prior to classification. In this way, information in the image is redistributed into a different 
2 tnd, depending on the application, more useful form. For example, transformations such as multi-spectral ratios of 
Near Infrared (NIR) to visible bands have been used to enhance reflectance differences between soils and vegetation 
and form “vegetation indices” that aid in classification. In this way, soil and other geological formations will exhibit 
similar ratios near 1, while vegetation will show a relatively larger ratio of 4 or more. Other common vegetation 
indices are the Normalized Difference Vegetation Index (NDVI), Soil- Adjusted Vegetation Index (SAVI), Transformed 
Vegetation Index (TVI), and the Perpendicular Vegetation Index (PVI).^ The success of using these indices in the 
past has been affected by relatively few acquisition dates during a growing season, the paucity of ground truth data at 
the time of acquisition, and the lack of suitable methods to account for atmospheric effects on the radiance measured 
by the remote sensing device.^ 

Whether a paxticul£u* classification algorithm uses spectral signatures or multi-spectral ratio indices to facilitate 
discrimination and detection of vegetation changes in an agricultural region, either technique requires good radio- 
metric calibration of the image before analysis can be performed. Figure 1 illustrates the major steps in the image 
classification process. Radiometric calibration, a fundamental stage in this process, generally involves (1) sensor 
calibration; at-sensor radiance values obtained from quantized data during A/D conversion, (2) atmospheric correc- 
tion: surface radiance values obtained from at-sensor radiance, and (3) solar and topographic correction: surface 
reflectance values obtained from surface radiance. Usually, det 2 uled information about atmospheric conditions is 
not available for a given data set. However, parametric atmospheric correction methods can generally be used to 
compensate for atmospheric conditions. The success of multi-image classification in the analysis stage is based on 
the quality of these methods. 

In this paper, we approach multi-image classification differently. Instead of using band ratios or absolute spectral 
signatures, we use “relative” signatures in the image to discriminate and detect vegetation changes. We compen- 
sate for the atmospheric effects on the multi- spectral images by applying the multi-scale retinex^ (MSR) image 
enhancement algorithm to the multi-image, prior to image classification. 

2. AGRONOMIC DATA 

For our analysis we used remote sensed images of a cotton field in Texas acquired in the summer of 1997. We chose 
two multi-spectral images of the field taken on consecutive dates. The first image (acquired 08/14/97) has overcast 


‘Senescense is the physiological death of plants. 









Figure 2. Test site captured on 08/14/97. The area of interest is highlighted within the rectangle. 

sky, i.e. diffuse light, and the second (acquired 08/15/97) has almost dear sky, i.e. direct sunlight. The highlighted 
sub-image in Figure 2 is the area of interest for this experiment. We label the images of this area TXoa for date 
08/14/97 and TXob for date 08/15/97. The cotton field was the site of a controlled experiment to study the effects 
on vegetation growth of different combinations of water and nitrogen treatment levels for a particular soil tillage 
type.^ In all, 4 water treatment levels, 5 nitrogen treatment levels, and 2 soil tillage types were used. Figure 3 shows 
a schematic of the treatment experiments applied to the TXoa and TXob regions. 

The field was divided into 120 blocks representing 4-5-2 = 40 unique combinations of water, nitrogen, and tillage 
type. Each combination was repeated 3 times over the whole field. The 4 irrigation levels used were: 0.00, 0.25, 0.50 
and 0.75 (fraction) of potentieJ evapotr^Lnspiration (PET)^. The 5 fertilizer nitrogen application levels were: 0, 20, 
40, 60 and 80 Ibs/acre Best Practice (BP)§. 

In theory, each of these 40 unique blocks represents a different “spectral class” and there axe three samples of 
each class. However, classification results show that the number of actual spectreil classes is fewer than 40. Moreover, 
the blocks are generally not homogeneous. That is, within each block there are mixed areas where the levels of water 
and/or nitrogen treatment are not uniform. In addition, the ground truth data was reliable for water treatment, but 

^Tillage prepares the soil for growing crops. This prepau*ation is traditionally accomplished by using a plow to cut and mix the soil. 

^Evapotranspiration (ET) is a measurement of the total amount of water needed to grow plants and crops. Since there are thousands 
of cultivated plants, the potential ET (PET) is a standard ET rate for general reference and use. The water requirements of specific 
crops and turf grasses can be calculated as a fraction of the PET. This “fraction” is called the crop coefficient (Kc) or turf coefficient 
(Tc)" 

5 Best Management Practices are farming practices that are designed to reduce nutrient contamination of surface and ground water. 
These practices are based on research results and field experiments zuid maybe as simple as following fertilizer recommendations and 
irrigation scheduling.^ 




4 Water Treatment Levels 
Wl- no irrigation 
W2- 25% PET (potential ET) 
W3- 50% PET 
W4- 75% PET 


5 Nitrogen Treatment Levels 
Nl- 0% BP (best practice) 

N2- 25% BP 20 Ibs/acre 

N3- 50% BP 40 Ibs/acre 

N4- 75% BP 60 Ibs/acre 

N5- 100% BP 80 Ibs/acre 


2 Tillage Types 
Tl- Bare Soil 

T2- Terminated Wheat 

3 Repetitions of each Tillage Type 


Figure 3. Agronomic treatments applied to the area of interest in Figure 2. 

not for nitrogen treatment. The four-band multi-spectral images were acquired from an aircraft platform with an 
approximate nadir view, and calibrated to reflectance. The aircraft multi-spectral sensor band centers were at 486, 
560, 685, and 840 nm. 

3. IMAGE PREPROCESSING 

3.1. The Multi-scale Retinex 

For all (x,y) pixels in the multi-spectral image G, the multi-scale retinex (MSR)®*^ can be compactly written as 

N 

n=l 

where the subscript j represents the spectral bands, N is the number of spati^d scales being used, and Wn are the 
weighting fgtetors for the scales.®’®’^ The i/„(x, y) are the surroimd functions (convolution kernels) given by 

Unix, y) = In exp[-(x^ + (2) 

where the (Tn are spatial scale parameters that control the extent of the surround function. Smaller values of an 
provide more dynamic range compression, and larger values provide more lightness /color constancy. The In are 
selected so that "^^Hn{x,y) = 1. Each of the expressions within the summation represents a single-scale retinex 
(SSR). 

The MSR combines the dynamic range compression of the small scale retinex with the tonal rendition of the large 
scale retinex to produce an output which encompasses both. Two fundamental issues in the application of the MSR 
are the following: 

1. The MSR reduces dependency on lighting conditions/geometry caused by such conditions as obsciued fore- 
grounds, and poor lighting caused by defects in illumination due to atmospheric conditions or artificial illumi- 
n2Uits. 

2. As atmospheric conditions change, the MSR will produce results such that the restored image / in Figure 1 
will look as much like the original scene s as possible, before image acquisition/digitization. 
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Figure 4. Post processing system model. 

4. DISCUSSION 

There were two primary motivations for this study: (a) to what extent can a conventional unsupervised classification 
algorithm yield “good” results when applied to the original images “as is” (i.e., with no preprocessing); and (b) if 
the multi-spectral images are preprocessed with the retinex algorithm and then the same conventional unsupervised 
classification algorithm applied, to what extent does that retinex processing influence the “goodness” of the result? 
This section summarizes the results of our initial experiments. 

4.1. MSR Pre-classification Processing 

The MSR was used to preprocess the multi-spectral image before it was used for classification. Figure 5 shows the 
results of the MSR algorithm applied to band 3 (685 nm) of the image in Figure 2. The left column shows the 
original images: TXoa and TXob, for the cloudy and clear day. The right column shows the MSR processed images: 
TXra and TXrb. In TXoa the band reflectance is uniformly very dark, because of the cloud cover on that day. The 
low contrast in this image creates a problem in obtaining spectral signatures that fidequately discriminate agronomic 
variables. After the image is processed with the MSR, subtle patterns emerged that were not visible in the TXoa 
image. Specifically, the patterns represent the boundaries between the 20 different nitrogen and water treatment 
regions. The MSR also improved the TXob image, obtained on the clear day. One of the primary results from the 
application of the MSR is that the processed images, TXra and TXrb, are more “similar” to each other in brightness, 
contrast and detail than the original images, TXoa and TXob. 

The MSR processed images display far more visual information than is evident in the unprocessed images. Even 
though radiometric calibration is not preserved by the MSR, we conclude that it can be used as an auxiliau-y tool for 
the visualization of spatial patterns in dark regions, as is demonstrated herein. Visual information in darker regions 
that may not be detected with linear representations which preserve radiometry will “pop out” with a clarity limited 
only to the dynamic range of the sensor and any intervening digitalization scheme used prior to the MSR. For this 
experiment, we have not yet conducted extensive performance comparison of the MSR with other image enhancement 
algorithms such as histogram equalization, gamma correction, and point logarithmic nonlinearity. However, we expect 
to find that those image enhancement algorithms ctre not appropriate for use in preprocessing multi-spectral images 
for remote sensing applications where atmospheric conditions are the major contributor to data inaccuracy. 

4.2. Multi-image Classification 

The literature is rich with both supervised and imsupervised methods for classifying remote sensed multi-spectral 
images. These methods include spatial /spectral discriminant functions, e.g., the Maximum Likelihood, and spectral 
specific methods, e.g., linear mixing models that require some a priori knowledge such as ground truth maps or 
ground samples. Whereas, supervised classification requires tradning sets to teach the classifier to recognize certain 
specific features in the image, unsupervised methods require little or no training data and attempt to discover the 
underlying patterns in multi-dimensional space by using techniques such as gradient descent. In this experiment, 
we have limited a priori information available about the field being analyzed. The a priori information was used to 
verify the accuracy of the classes obtained using unsupervised classification. 

We used vector quantization (VQ) to perform unsupervised classification on the multi-spectral image. The only 
user specified parameter is the number of classes K. The primary goal was to study and compare the effects of MSR 






08/14/97 TXoa 08/14/97 TXra 



08/15/97 TXob 08/15/97 TXrb 


Figure 5. TX Images- Original and retinex processed images. 




Table 1. List of constants used to process the TXoa and TXob images with the MSR 


Constant 

(Tl 


CTZ 

CT4 

Gain 

Offset 

Value 

2 

5 

20 

200 

180 

0.57 


preprocessing on the classification results. The same classification algorithm was applied to both of the original four- 
band images TXoa, TXob and both of the MSR four-band images TXra, TXrb. For ail four images, we systematically 
experimented with K to see how the number of classes affects the overall classification results. 

To cluster the images we used VQ along with a splitting method to define the spectral signatures. The method 
starts with a one-level quantizer (i.e., the centroid of the entire training set). Next, the one-level quantizer vector 
is split into two vectors obtained by perturbing the one-level quantizer. The 2-level quantizer is then applied to the 
training set. The two 2-level quantizer vectors are then split into four vectors and a 4-level quantizer is applied to 
the training set. The splitting is continued in this manner until K code vectors are generated. This method assumes 
that K is a power of two. If K is not a power of two, then in the last step, instead of generating two new code vectors 
from each of the code vectors of the quantizer designed previously, we can perturb as many code vectors as necessary 
to obtain the desired number of code vectors.^® As in most classification methods, the performance depends on the 
quality of the set of spectral means used to discriminate classes in the image. For this analysis, we did not focus on 
methods to obtain spectral means, but compeired the relative accuracy of the spectral means obtained by the VQ to 
signatures derived from the training areas defined by the schematic map of Figure 3. 

5. CLASSIFICATION RESULTS 

Figures 6 and 7 show the classification results for the original and MSR preprocessed images. To examine the accurzwry 
of our results we compare these results to the schematic plot of the proposed treatment of water and nitrogen for 
the cotton field shown in Figure 3. To facilitate analysis, the figures are annotated with a grid that provides an 
approximate separation boundary between each treatment level block. 

Our classification results were very encouraging for a number of classes, from as few as K = 4 — (i.e., 4 water 
treatment levels), to as many as K = 40 — (i.e., 4 water treatment levels x 5 nitrogen treatment levels x 2 tillage 
types). For the case K = 4, we were interested in determining how well the four water treatment levels could be 
discriminated in the image. In the case AT = 8, we were interesting in discriminating the four water treatment levels 
for each tillage type. FVora our results we did not see any major differences between the different tillage types in 
terms of classification results. That is to say, we could not resolve two different classes of each water treatment type. 
The primary effect of the case K = 8 was that we were able to see more clearly the water-nitrogen iterations. 

Generally, for all four images we were able to see very clearly the “block” treatment structure that is present. 
There are differences in the results, however, depending on whether the lighting was diffuse or direct and depending 
on whether or not retinex preprocessing was used. The left column of Figures 6 and 7 show that without MSR 
preprocessing the blocks tend to be classified as homogeneous (one class); the right column of Figures 6 and 7 show 
that with MSR preprocessing the blocks tend to be classified as more non-homogeneous (multi-cleiss) . Comparison 
of the classification results for TXoa, TXob (original images) and TXra, TXrb (MSR processed images), show 
considerable variation in classes for the unprocessed images as the atmospheric and lighting conditions vary, but 
slight or no variations for the MSR processed images. 

Because consistent classification results are adiieved regardless of the atmospheric conditions, we can argue 
empirically that MSR preprocessing tends to produce “spectral signature images” . Note that classification consistency 
in this experiment is really a measure of the resiliency of the classification process to changes in the process that 
affect the formation of the multi-spectral image. In other words, classification consistency is really a measure of how 
well we can cleissify the multi-spectral image given that the atmospheric conditions have changed substantially from 
day to day. To illustrate classification consistency, in Table 2 we show the mean spectral reflectance measurements 
for the original and MSR preprocessed images for each class. From this table we see that the spectral signatures for 
the MSR preprocessed images for each day are more similar than the signatures for the original images. 

Initially, we expected to see one different class for each column in the field representing a different water treatment 
level and not the multi-class variations that are shown in Figures 6 and 7. However, because the nitrogen and water 



■ 

■_E_£a 


■ nr 

■in 


m mi 

mmm 



Kmu 

Darker 

Reflectance 



B 

in W3 

08/14/97 TXoa 

08/14/97 TXra 



■ W2 

B 

Bj^Ba 

W1 

B ma 

mwwn 

Lighter 

Reflectance 

m MM 

BXail 



Bii«BS 


B 

5® 

B 

B 

BSVJil 


08/15/97 TXob 

08/15/97 TXrb 



Figure 6. Classification Results: 4 Classes 






• • 











2. Mean spectral reflectance measurements for each class. 

I 11 ■ 


4 Sfi — ^ Iwavelength (nm) - 

0871479TMSR 


55 

58 

60 

63 


49 

52 

54 

58 


33 

38 

41 

45 


102 

93 

104 

90 


08/15/97 direct 
115 113 66 201 

127 128 83 195 

136 138 97 190 

148 149 106 190 


140 138 123 161 

146 145 139 155 

151 148 151 146 

158 157 167 144 

08/15/97 MSR 
124 119 92 156 

143 140 126 152 

146 148 159 149 

169 167 173 147 




486 

54.83 
60.37 
56.77 
58.29 

102.07 

124.47 

112.83 
117.06 


560 685 

ua/ 14/97 diffuse 
50.68 33.49 

54.88 41.54 

52.15 36.85 

53.42 39.12 

08/15/97 direct 
104.53 61.38 

126.46 91.42 

115.05 76.03 
120.30 83.67 


103.65 

91.22 

94.85 

92.49 

176.28 

162.21 

165.88 

162.03 


140.46 
154.25 
143.66 
148.19 

128.47 
157.97 
141.24 
147.88 


560 685 

^«/ 14/97 MSR 
141.93 125.12 

153.09 156.12 

143.80 134.99 

147.92 144.54 

08/15/97 MSR 
127.800 101.82 

157.38 164.08 

140.29 130.38 

147.31 145.53 


840 

162.60 

145.28 

153.09 

150.00 

156.43 

146.14 

151.37 

149.41 


Class 


W4 

Wl 

W3 

W2 

W4 

Wl 

W3 

W2 


increased with water <!trp«c t n ^^ter treatment, i.e., joint effects. In Tahio o S s the change in reflectance 

840 nm ranee Thic • ^ ''^ater levels in the 486 560 and 68^ ^ reflecttince generally 

resists Z 

for em* dass^T^„“ wit 

clai^shoti^'^fo Tabl^^^^ processed measurements of each column 

infonm..“ ^7'“ Pr«en.ed he^fsElS 

^expected from pa rticular agronomic variable interaSiZ the amount of 

is mems a reexamination of the data, which is the subiec. fmw eu atmospheric conditions. 



6. CONCLUSIONS 

Spectral signatures alone do not provide adequate classification of a scene, especially if the atmospheric or lighting 
effects have severely affected the multi-image components.^ This is evident if we compare the classification results 
before and after MSR preprocessing. Without additional ground truth, or results from other classification studies, 
it is difficult to state with any confidence whether the classifications obtained with the preprocessed images are 
“better” in some absolute sense than the classifications obtained from the original images. We can state, however, 
that the classifications from the MSR preprocessed images for the two different lighting and atmospheric conditions 
are remarkably “similar” both visually, and in terms of the mean spectral reflectance of a class. We speculate that this 
occurs because the MSR preprocessing is minimizing the effects of the atmospheric conditions on the multi-spectral 
image, leading to consistent classifications from consistent data. 

To summarize, we conclude that conventional unsupervised classification can be applied to this significant problem 
of detection and discrimination of stressed and imstressed vegetation. Although classification results from both the 
original ctnd the MSR preprocessed images are encouraging, the MSR preprocessed images are more robust to 
changes in atmospheric and lighting conditions. We need to conduct additional experiments to test the validity of 
our speculation that MSR preprocessed multi-image classification is more robust in the presence of atmospheric and 
lighting changes. In addition, we need to substantiate our conjecture that other image enhancement algorithms do 
not have the same “beneficial” effect on the classifications. A color version of the figures in this paper is available at 
http://dragon.larc.nasa.gov/viplab/retinex/background/pubabs/spie3716-1999.html. 
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ABSTRACT 

Image preprocessing is useful in helping to identify “spectral response patterns” for certain types of image classifi- 
cation problems. The common artifacts in remotely sensed images are caused by the blurring due to the optics of 
the image gathering device, illumination variations, and the radiative transfer of the atmosphere. The Multi-Scale 
Retinex (MSR) image enhancement adgorithm that provides dyn2unic range compression, reduced dependence on 
lighting conditions, and improved (perceived) spatial resolution has proven to be an effective tool in the correc- 
tion of image degradations such as those in remote sensing images. In this paper, we measure the improvement 
in classification accuracy due to the application of the MSR algorithm. We use simulated images generated with 
different scene irradiance and with known ground truth data. The simulation results show that, despite the degree 
of image degradation due to changes in atmospheric irradiance, classification error can be substantially reduced by 
preprocessing the image data with the MSR. Furthermore we show that, similar to the results achieved in previous 
work, the classification results obtained from the MSR preprocessed images for various scene irradiance are more 
similar to each other than are the classification results for the original unprocessed images. This is evident in the 
observed visual quality of the MSR enhanced images even before classification is performed, and in the difference 
images obtained by comparing image data under different irradiance conditions. We conclude that the application 
of the MSR adgorithm results in improved visual quality and increased spatial variation of multispectraJ images that 
is also optimal for certain types of multispectral image classification. 

Keywords; image classification, image enhancement, multispectral, retinex, dynamic range compression 

1. INTRODUCTION 

Various algorithms for image classification using remotely sensed imagery exist in the literature. One way to charac- 
terize these algorithms is by the characteristics of the multi-dimensional space in which they operate. For example, 
many users of remotely sensed imagery utilize spectral signatures to characterize and identify materials in multi- 
dimensional “spectral” space. The spectral signature of a material may be defined in the solar-reflective region of the 
electromagnetic spectrum by its reflectance as a function of wavelength, measured at an appropriate spectral reso- 
lution. In other spectral regions, signatures of interest are temperature and emissivity (TIR) and surface roughness 
(radar). The motivation for using remote sensed data for material identification is that different types of materi2ils 
exhibit different spectral signatures,^ and so can be distinguished on this basis. 

There are fundamental problems with the spectral signature approach that are well documented in the literature. 
One fundamental problem is that all spectral signatures are unique to the sample and to the environment in which 
they are obtained. Further, the ability to distinguish spectral signatures is often complicated by the natural variability 
of a material, the spectr2d qu£Uitization of many remote-sensing instruments, and the modulation of the signatures 
by the atmosphere in the image formation process.^ Therefore, there is no guarantee that the spectral signatures 
obtained by the remote sensing system will either be similar to the ones obtained under a different environment, or 
exhibit measurably different, or even recognizable characteristics. 

In recent years, considerable quantities of ground-based (laboratory) data have been accumulated that describe 
spectral reflectance characteristics of several types of soils and vegetation. However, it is virtually impossible to 
duplicate natural reflectance variations under laboratory conditions. In addition, the spectral signature of vegetation 
changes over the seasonal life cycle of plants. Thus the comparison between natural reflectance signatures and 
laboratory produced signatures becomes even more complicated. 
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Figure 1. System Model. 


As an 2 ilternative to classification based on spectral signatures, the multi-dimensional spectral space can be 
transformed into an application dependent “feature” space that may prove more useful for classification purposes. 
For example, transformations such as raulti-spectraJ ratios have been used to enhance reflectance differences between 
types of soil and vegetation, aind are used to form “vegetation indices” that aid in classification. TypiceJly, soil and 
other geological formations exhibit ratios near 1, while vegetation shows a ratio of 4 or more. Other transformations 
such as scaJe space filtering have been applied to identify “fingerprints” of certain types of minerals using local points 
of inflection to characterize absorption characteristics. The success of using these indices has been greatly affected 
by the lack of suitable methods to account for atmospheric effects on the radiance measured by the remote sensing 
device.*^ 

Figure 1 illustrates the major steps that relate the image acquisition to the image classification process. Regatrdless 
of which feature classification method is to be used, the (raw) image data needs to be radiometrically calibrated before 
it can be used for analysis. That is, the raw data must be converted from sensor DN values to surface reflectance 
values. Radiometric calibration generally involves several steps: sensor calibration — calculation of gain and offset 
coefficients that convert sensor DN values to at-sensor radiance values; atmospheric correction— con verion of at- 
sensor radi^ulce values to surface radiance using atmospheric modeling and estimation correction techniques; and, 
solar and topographic correction — conversion of surface radiance to surfetce reflectance by correcting for topographic 
slope and aspect, solar spectral irradiance, solar path transmittance, and down-scattered “skylight” radiance,^ In 
addition, detailed information about atmospheric conditions at the time of data acquisition may be required but is 
generally not available. Parametric atmospheric correction methods can also be used to compensate for atmospheric 
conditions, but they also require some information about the atmospheric conditions at the time of data acquisition. 
The success of multi-image classification in the analysis stage is, thus, dependent on the quality of these calibration 
and correction methods. 

In this paper, we approach multi-image classification differently. Instead of applying (parametric) atmospheric 
correction methods to remote sensed imagery, we compensate for the atmospheric effects by applying the multi-scale 
retinex (MSR) image enhancement algorithm to the multispectral data prior to classification. The d 5 mamic range 
compression and color constancy properties of the MSR aid in minimizing the effects of variations in illumination 
conditions, and the sharpening compensates for the device blurring and atmospheric amplitude modulation. 

2. IMAGE PREPROCESSING 

There are many factors that contribute to degrade the acquired image. For example, the device signal-to-noise ratio 
(SNR) and the blurring due to the point spre 2 td function (PSF) of its optics, and the quantization artifacts due to 
the analog-to-digital converters are produced by the image acquisition device. In addition, platform perturbations, 
atmospheric modulations, and sampling artifacts also degrade the acquired image. Most researchers agree that 
geometric and radiometric artifacts are the most common cause of image degradations in remotely sensed imagery.^ 
With reference to Figure 1 image restoration is an attempt to make the restored image g be geometrically and 
radiometrically as “close” as possible to the radiant energy characteristics of the original scene s. The closeness 
is measured in some metric space, such as the minimum mean square error (MSRE) space, and the goal of the 
restoration process is to minimize the MSRE between the restored image and the original scene. Although, generally 
termed an image enhancement technique, the MSR has proven to be an effective technique for correcting image 







Figure 2. 2-d Mondrian. 


degradations due to the optical blurring of the image acquisition device, illumination variations, and atmospheric 
modulation. Thus, the MSR can be used to “restore” the acquired data, without any prior knowledge about the 
atmospheric conditions at the time of acquisition. 

2.1. The Multi-scale Retinex 

For all (x, 2 /) pixels in the multi-spectr^Ll image G, the multi-scale retinex (MSR)**'^ can be compactly written as 

N 

-log[Gj(i,y)*H„(x,j/)]}, i = (1) 

n=l 

where J represents the number of spectral bands, N is the number of spatial scales being used, auid are the 
weighting factors for the scales.®”® The Hn{x^y) are the surround functions (convolution kernels) given by 

Hn{x, y) = /„ exp[-(i2 + (2) 

where (Tjy, are the spatial scale parameters that control the extent of the surround function and the In are selected 
so that 2 ^ Hn{x,y) = 1. Smaller values of an provide more dynamic range compression, and larger values provide 
more lightness/ color rendition. Each of the expressions within the summation represents a single-scale retinex (SSR). 

The MSR combines the dynamic range compression of the small scale retinex with the tonal rendition of the 
large scale retinex to produce an output which encompasses both. The MSR reduces dependency on lighting con- 
ditions/geometry caused by such conditions as obscured foregrounds, and poor lighting caused by atmospheric 
conditions or defects in artificial illuminants. 

3. SCENE GENERATION 

In order to exactly measure the effectiveness of MSR preprocessing on multi-image classification, we use simulated 
images with exsictly known ground truth. The simulated images are created by combining known atmospheric 
transmittance profiles with scenes with known mean spatial detail and surface reflectance. We use a simple modeP’^ 
of the radiance field which has the following char 2 urteristics: 

1. The scene is a two-dimensional Mondrian flat surface divided into patches of uniform reflectance. 

2- The effective irradiance (or the atmospheric transmittance) /(x,y) varies slowly and smoothly across the entire 
scene, cind 

3. The reflected radiance field L(x,y) is everywhere independent of the viewer^s position. 

These assumptions permit us to express the radiance field L(x, y) by the simple relationship 

= ^P(a:,2/)/(x,y), 




(3) 




Figure 3. Scene generation system model. 

where p{x^y) is the Lambertian surface reflectance. The target scene with reflectance p{x^y) is represented by the 
two-dimensional (2-d) Mondrian illustrated in Figure 2. This scene consists of random polygons whose boimdaries are 
Poisson distributed and whose reflectances are distributed according to independent zero-mean Gaussian statistics. 

The scene used in the simulation has 256 x 256 pixels. The reference spectra (See Section 3.1) were subsampled 
at every 0.05^m to produce ideal spectra of 42 points. This results in a multi-image with of 256 x 256 x 42 values. 
Figure 3 illustrates the process by which the 42 band veal multi-image is generated. The multi-image is generated 
by assigning each spatial location in the Mondrian a number corresponding to a speciflc reference spectra that 
corresponds to the identification number for a vegetation type. 

The atmospheric transmittauice function is simulated by significantly blurring a generated Mondrian image. In 
addition to providing regions of different transmission characteristics, this also simulates the umbra and penumbra 
profiles of shadows. Two instemces of the atmospheric transmittance profile are shown in Figure 4: the dark regions 
represent absorption in the atmosphere, or clouds; and the light regions represent tr 2 insmittance, or absence of clouds. 
The first profile, atransl, has two moderately sized regions of low transmittance {< 30%) at the lower left and right 
comers of the image. For the majority of the scene the transmittance is about 45%, with some higher transmittance 
areas located at the bottom center portion of the image which have a transmittance of about 80%. The second 
profile, atrans2, has on average a transmittance of about 65%. As shown in Figure 3, after the ideal scene image m 
and atmospheric transmittance profile m' have been generated, the simulated multi-image g is obtained by doing a 
pixel by pixel multiplication of m' and m, 

g{x,y) =m{x,y)-m'{x,y) 

Finally, the real image g is processed with the MSR algorithm to create the processed image /. 

In order to analyze the data, fidelity metrics are computed for the ideal, real, and the MSR processed images. 
Because we have ‘Aground tmth” data, these fidelity metrics can be applied ^tnd the results compared to the ground 
truth. The fidelity analysis is based on two metrics that measure the accuracy and consistency of the results as they 
are affected by the application of the two atmospheric transmitt 2 Uice profiles. A mean squared-error metric is used 
to measure the similarity between the two original images, and the similarity between the two real retinex images 
for each atmospheric transmittance profile. A sensitivity metric is used to measure classification consistency and 
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Figure 5. Reference spectra of vegetation and soil types. (Source: USGS Speclab.^®) 



Figure 6. real 2-d Mondrian scene: (1) originaJ (r) linear contrsist stretched version. 




Figure 7. real 2-d Mondrian scene: (1) MSR processed (r) linear contrast stretched version. 









Or^VDil atnoi 1 


Ofipnil •tnni2 



Retim itmifl RefinBi >tnni2 


Figure 8. Effect of atmospheric models on original and MSR processed images. 


the original atransl image. Very bright areas of reflectance in the original atrans2 image axe aJso discernible in areas 
where the transmittance was the highest. In the MSR atransl image there is some slight darkening of reflectance 
in the lower left corner of the processed image, however, overall the MSR images are more similar to each other 
than are the unprocessed images. We also observe that the boundaries between regions in the MSR images are more 
clear and are in greater detail than the original. The previously identified edge artifacts ctre, however, evident at the 
transition between dark and bright areas in the processed images. 

5. CLASSIFICATION 

We used vector quantization (VQ) to perform unsupervised classification on the multi-spectral image. The only user 
specified pareuneter is the number of classes K. For classification, a 9 band subset of features were chosen from the 
original 42 band image. 

To cluster the images we used VQ along with a splitting method to define the spectral signatures. The splitting 
algorithm used to generate the trained codebook, splits each training set codebook vector using the best perturbation 
factor for that dataset. The preferred perturbation factor is the one that generates the smaJlest MSE for the input 
training set. The algorithm is designed to produce cluster means for a specific codebook size. As is the case for most 
cle^sification methods, the performance depends on the quality of the set of spectral means used to discriminate 
classes in the image. For this analysis, we did not focus on methods to obtain spectral means, but compared the 
relative accuracy of the spectr 2 d means obtained by the VQ to signatures derived from the training areas 

Because we have a ground truth map of our ideal classification, the training set will be selected from regions 
in the image shown in Figure 2. The training set vectors were input into the splitting algorithm and the trained 
codebook vectors were generated. In the testing stage, the images were classified with the trained codebook vectors, 
using a MSE VQ clustering algorithm. The resulting test codebook vectors were used as candidate spectral means 
to identify each vegetation species. 
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Figure 10. Classification results for original and MSR — No Atmospheric Models Applied. 


Figure 9 shows the classification results for the original and MSR imc^es for the two transmittance profiles. The 
classification results for the MSR processed images axe more consistent with each other than the results obtained 
with the original images. However, there were problems separating certain class pairs such as pasture (c) and potato 
(nh), and barley and bare field. The pasture (c) spectra is not identified in the atrans 2 model image for the original. 
Both the original and the MSR processed atrans 1 images have classification errors resulting from the affect of the 
low transmittance area in the lower left portion of the images. 

Tables 1 and 2 show the classification sensitivity results for the original and MSR image for different ground 
truth images. The original and MSR ground truth images refer to the images shown in Figure 10. The sensitivity 
measurements are listed in terms of the percentage of pixels correctly classified and the actual number of pixels 
correctly classified out of the total 65536 (256 x 256) pixels that belong to the classified image. As can be seen in 
Table 1, the MSR processed image for the atrans 1 transmittance profile provided better classification sensitivity. 
When used with the real original ground truth image, the MSR produced sensitivity results slightly better, 2.5%, 




than those produced by using the original atransl data. However original image for the atrans 1 model. However, 
when the MSR processed original data was used for training, the classification accuracy for the MSR processed image 
was substantially better, 15%, than the original. An even greater sepeu*ation in results is seen between the original 
and MSR images for the atrans2 transmittance profile. When the MSR processed data is used for training, the MSR 
classified image is approximately 23% better than the original classification. Even when the original unprocessed 
data is used for training, the MSR classification is about 10% better. 


Table 1. Sensitivity (# pixels out of 65536) 


Ground Truth 

Image 

Model 1 



% correct 

# pixels correct 

Real Original 

Original 

57.56 

37722 

Real Original 

MSR 

60.99 

39970 

Real MSR 

MSR 

70.63 

46288 


Table 2. Sensitivity (# pixels out of 65536) 


Ground Truth 

Image 

Model 2 



% correct 

# pixels correct 

Real Original 

Original 

54.32 

35599 

Real Original 

MSR 

74.34 

48719 

Real MSR 

MSR 

86.86 

56924 


Figure 11 shows an RGB composite of the quantized images. The original images have been linearly stretched so 
that subtle differences between them and the MSR image can be compared. Comparing the effects of the atmospheric 
models on the original and MSR images, we see that the MSR images appear visually consistent between both models 
which is consistent with the results obtained using the 42 band multispectral image results discussed earlier in this 
experiment. However, the effects of the transmittauice models on the original images is more apparent in these 
quantized images. The low transmittance areas are sharply contrasted with the high transmittance regions. In the 
MSR atransl image we do observe the same darkening of reflectance in the lower left comer of the image which 
corresponds to the same low transmittance in the atransl model in the same area. We observe the same edge 
artifacts effects around the borders of regions in the MSR quantized images as seen in the 42 band MSR image. For 
this experiment, the advantage of generating classification and quantized images simultaneously is that the classified 
image may provide an indication of how closely the quantized image will match the original data. 

In Figure 13 we illustrate the results of using the squared-error difference metric to compare the original and 
MSR images for each atmospheric transmittance model. We conclude from the difference images that the consistency 
observation between the MSR images for the difference transmittance models is again confirmed. These results are 
similar to the results shown in Figure 12. However, we observe more high difference regions in Figure 13 for the 
original image, than in Figure 12. 


6. CONCLUSIONS 

Although image enhancement is typically applied to improve the visual quality of multispectral images, in this ex- 
periment we have given quantitative evidence that the application of the MSR algorithm restores images that are 
degraded by atmospheric transmittance effects, and improves the results of multispectral image classification. Be- 
cause the MSR 2 dgorithm was applied before clustering, the classification algorithm generated candidate spectra that 
were better sepeu-ated in reflectance for the MSR images than the spectra generated for the original images. Further- 
more, the MSR ceindidate spectra maintained separability and high reflectance values reg£U*dless of the atmospheric 
transmittance models applied. This leads us to conclude that the application of the MSR algorithm produces (ap- 
proximately) illuminant invariant spectral signature images. Except for class regions in which edge artifacts produced 
incorrect classifications abound region boundciries, the classification results and the difference measurement results 
show a consistency between MSR images that is not evident in the classifications based on the original images. 
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Figure 11. Effect of atmospheric models on VQ original and MSR images. 



Figure 12. Difference images for the (1) original and (r) MSR images. 





Figure 13. VQ Difference images for the (1) original and (r) MSR images. 
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